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Abstract 

We consider a public multi-user information embedding (watermarking) system in which two mes- 
sages (watermarks) are independently embedded into two correlated covertexts and are transmitted 
through a multiple-access attack channel. The tradeoff between the achievable embedding rates and 
the average distortions for the two embedders is studied. For given distortion levels, inner and outer 
bounds for the embedding capacity region are obtained in single-letter form. Tighter bounds are also 
given for independent covertexts. 



Index Terms: Capacity region, correlated covertexts, multiple access attack, multi-user information embed- 
ding, inner and outer bounds, public watermarking. 

1 Introduction 

In the last decade, the single-user (point-to-point) information-hiding (information-embedding, watermark- 
ing) model has been thoroughly studied from an information-theoretic point of view; see, e.g., |[T1 l9l [TSll 
and the references therein. With the rapid development of wired and wireless communication networks, 
situations arise where privacy protection is no longer a point-to-point problem. Therefore, it is of interest to 
study information-hiding problems in multi-user settings. 

In this paper we consider the scenario in which two secret messages (watermarks) are independently 
embedded in two correlated sources (covertexts) and are then jointly decoded under multiple-access attacks. 
This scenario is motivated by, for example, the practical situation where audio and video frames are wa- 
termarked separately, but they are transmitted in a single bit stream and decoded by one multimedia player 
(see ifTOl [121 HI). The model is depicted in Fig. [T] and it assumes that two users separately embed their 
watermarks Wi and W2 into two correlated discrete memoryless sources (DMSs), Ui and U2- Each user 
can only access one of the two covertexts. The watermarked messages (stegotexts) and X2 are then 
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sent through a multiple-access attack channel (MAAC) to a decoder which attempts to reconstruct the wa- 
termarks. For this two-user information embedding system we are interested in determining the embedding 
capacity region; i.e., the two-dimensional set of all achievable embedding rate pairs under constraints on the 
embedding distortions. 
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Figure 1 : A multi-user information embedding system with two embedders. 



Our main result (Theorem is an inner bound for the embedding capacity region. The proof is based 
on the approach of Gelfand and Pinsker ||3 and a strong typicality coding/decoding argument. The encoders 
first map the watermarks Wi and W2 and the correlated covertexts [/" and U2 to auxiliary codewords T" 
and T2, and then generate two stegotexts and X2 which are jointly typical with ([/", f/l*, T", T|*). 
The decoder recovers the watermarks by examining the joint typicality of the received sequence and all 
auxiliary codeword pairs (T", T|*). 

One major technical difficulty is the problem of how to separately construct the typical sequence en- 
coders. In order to guarantee that the codewords together with the covertexts are jointly typical with a high 
probability, we adopt a "Markov" encoding scheme from (TT\, which was originally proposed for Gaussian 
multi-terminal source coding (see also 1 13 1 and |6|). The Markov encoders can be briefly described as fol- 
lows. One of the encoders (embedders), say Encoder 1, first forms an estimate of the source sequence of the 
other encoder, and then generates T" which is jointly typical with the observed source sequence [/" and the 
estimated source sequence. The other encoder. Encoder 2, first forms an estimate of the source sequence as 
well as the auxiliary codeword of Encoder 1, and then generates T2 which is jointly typical with the source 
sequence U2 and all the other sequences estimated. For the resulting scheme, an extended Markov lemma 
(Lemma ID ensures that the auxiliary codewords Tf and T2, although generated by separate encoders, are 
jointly typical with the source sequences with a high probability. 

We also derive an outer bound for the embedding capacity region with single-letter characterization 
(Theorem using Fano's inequality and a standard information-theoretical bounding argument. We spe- 
cialize the embedding capacity region to independent covertexts and obtain inner and outer bounds for this 
case (Theorem O. The inner bound is a consequence of Theorem [T] while in the converse part we sharpen 
the bound of Theorem [2]by making use of the independence condition. 
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We note that the multi-user information embedding problem studied in this paper is related to the works 
|[T2l and lUl. In |[T2l . the authors present an achievable embedding region for correlated Gaussian covertexts 
and parallel (independent) additive Gaussian attack channels (as opposed to the MAAC considered here). 
In a recent work [8|, the authors study the same system as ours and give an inner bound for the capacity 
region without a proof, stating that this inner bound can be easily proved via the coding procedure in lfT2l . 
However, the proof in |[T2l seems to be incorrect because the encoders cannot guarantee the typicality of 
the output sequences with respect to the covertexts sequences. Our code construction corrects this problem 
and in Theorem [T] we show that the main result in lfT2l (the achievable region) and the inner bound given in 
||8 1 are both correct. We also point out that a similar setup concerning a multi-user reversible information 
embedding system was considered in |7 | and 18 ] for two covertexts and a MAAC. Since in the reversible in- 
formation embedding problem the secret messages and the covertexts are both reconstructed at the decoder, 
Gelfand and Pinsker coding is not required and the coding strategy is fundamentally different from ours. 

The remainder of this paper is organized as follows. We set up the public multi-user embedding (wa- 
termarking) problem, define the embedding capacity region, and present our main results in Section |2] The 
proof of the inner bound is given in Section [3l while the proof of the outer bounds are deferred to the 
Appendix. We close the paper with concluding remarks in Section HI 

2 Problem Formulation and Main Results 

Let I X I denote the size of a finite set X. If X is a random variable (RV) with distribution Px , we denote its n- 

(n) 

dimensional product distribution by Pj^ . Similar notation applies to joint and conditional distributions. For 
RVs X, Y, and Z with joint distribution Pxyz, we use Px, PxY, Pyz\x^ ^tc, to denote the corresponding 
marginal and conditional probabilities induced by Pxyz- The expectation of the RV X is denoted by E(X). 
All alphabets are finite, and all logarithms and exponentials are in base 2. 

Let Ui and C/2 be two discrete memoryless host sources with alphabets Ui and U2 and joint dis- 
tribution QuiU2- The watermarks Wi and W2 are independently and uniformly chosen from the sets 
>Vi = {1,2,..., Ml} and W2 — {1,2,..., M2}, respectively. The attack channel is modeled as a two-sender 
one-receiver discrete memoryless MAAC Wy\XiX2 having input alphabets and X2, output alphabet y, 
and transition probabihty distribution WY\XiX2iy\^i^^'2)- The probability of receiving y € 3^" conditioned 
on sending xi G A"" and X2 G X2 is hence given by W^^-^^-^^ (y|xi, X2). 

Let di : Ui X Xi — > [0, 00) be single-letter distortion measures and define d™"^ = maxdj(uj, Xj) for 
i = 1, 2. For Ui G UJ" and Xj G A^, let di{ui,Xi) = J2]=i di{uij,Xij). 

A two-sender one-receiver multiple-access embedding (MAE) code {f["\ f^\'^^"'^) with block length 
n consists of (see Fig. [D two encoders (embedders) 

/{"^ : Wi X X"^ and /^"^ : W2 x Ul^ X^ 

with embedding rates Rfi = ^ log2 Mi and Rj^ = ^ log2 M2, respectively, and a decoder 

^{n) ,yn ^y^^^ y^^_ 

The system depicts a "public" embedding scenario since the covertexts are not available at the decoder. 
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The probabiUty of erroneously decoding the secret messages is given by 

Ml M2 

where Xj = f^^\wi,Ui) for i = 1, 2. 

Definition 1 Given Qc/iC/j, Wy\XiX2^ ^ ''^^ pair R2) is said to be achievable with respect to distortion 
levels {Di,D2) if there exists a sequence of MAE codes {f'f'\ f2^\'4^'^"'^) at embedding rates no smaller 

(n) 

than Ri and R2, respectively, such that lim„^oo =0 and 



lim sup — E 



n— >oo 



<A, i = L2. 



The embedding capacity region TZ{Di,D2) is the closure of the set of all achievable rate pairs R2). 

Remark 1 It can be shown by using a time-sharing argument H that Tl{Di,D2) is convex. 

Definition 2 Given QuiU2' ^y\XiX2^ apair of distortion levels {Di, D2), let Sdi,D2 be the set of RVs 
(C/i , Ti , C/2 , r2 , Xi , X2 , y ) G ^/i X 71 X X Ti X Afi X ^"2 X 3^ for some finite alphabets 71 and Ti such that 
the joint distribution PU1T1U2T2X1X2Y satisfies: (1) PU1T1U2T2X1X2Y = Qc/iC/z^TiXili/i-fraXalc/a^yiXiXa^ 
(2) /(C/^; r,) > 0, and (3) E[diiU^, Xi)] < Di, for i = 1,2. 

Definitions Given Q^^f/j, Wy\XiX2^ and a pair of distortion levels {Di, D2), 1^1^01,02 be the set of RVs 
(C/i , Ti , C/2 , r2 , Xi , X2 , y ) G Wi X 71 X ^^2 X 72 X A'l X ^2 X 3^ for some finite alphabets 71 and T2 such that 
the joint distribution PuiTiU2T2XiX2Y satisfies: (1) PU1T1U2T2X1X2Y = Qc/iC/a^TiTaXiXalc/iC/a^yiXiXa^ 
and (2) E[d^{Ui,Xi)] < Di, for i = 1, 2. 

Note that the only difference between the two regions is that in the definition of Sdi,D2' the conditional 
distribution of (Ti, T2, Xi, X2) given {Ui,U2) is restricted to be in the form PtiXi\UiPt2X2\U2- This of 
course implies Sdi,D2 ^ 'Pdi,D2- 

The following are the main results of the paper. 

Theorem 1 (Inner bound) Let TZin{Di, D2) be the closure of the convex hull of all R2) satisfying 

i?i < I{Tr,T2,Y)-I{Ui;T^), (1) 
R2 < I{T2;Ti,Y)-I{U2;T2), (2) 
i?i+i?2 < I{Ti,T2;Y)-I{Ui,U2;n,T2), (3) 

for some (C/i, Ti, [/2, Ta, Xi, X2, y) G Then 7^,„(L>l, L>2) ^ ^(1^1, 1?2). 

The proof of the theorem is given in Section [3] 

Remark 2 As we show in Appendix O the cardinality of the alphabets of the auxiliary RVs Ti and T2 for 
T^in{Di, D2) can be bounded as \Ti\ < \Ui\\U2\\Xi\ + 1, i = 1,2. 
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Remark 3 Although we only deal with discrete (finite-alphabet) sources and channels, it is not hard to 
see that, with the appropriate changes in the proof, the achievable region is also valid for a system that 
incorporates a pair of correlated memoryless Gaussian sources and a Gaussian MAAC. In particular, when 
the MAAC is a pair of parallel (independent) additive Gaussian channels, Tlin{Di, D2) is the achievable 
region obtained in IIT2I . even though the proof provided in [12] is not entirely correct. Note also that our 
inner bound TZin{Di, D2) is the same as the one given without proof in [8, Proposition 1]. 

Theorem 2 (Outer bound) Let lZout{Di, D2) be the closure of the collection of all rate pairs {Ri, R2) sat- 
isfying conditions ([B-© for some (C/i, Ti, C/2, Ts, Xi, X2, € Vd^,D2- Then 7^(L>l, L>2) ^ 7^o„^(L»l + 
5, D2 + 6) for all 6 > 0. 

The proof of the theorem is given in Appendix |Al The proof involves Fano's inequality and a (by 
now) rather standard information-theoretic argument that generalizes the converse proof for a single-user 
embedding system in lil5J . 

Remark 4 The above theorem states that TZ{Di, D2) ^ ^5>q '^out{Di + 6, D2 + 6). If we could upper 
bound the cardinality of the alphabet sizes of the auxiliary RVs Ti and T2 in the definition of TZout{Di, D2), 
it would be easy to show that f^^^QTZoutiDi + 6, D2 + S) = Tlout{Di, D2), so that Tl{Di,D2) C 
T^out{Di, L>2). However, without such an upper bound, we can only state the theorem in the present weaker 
form. The same remark applies to the outer bound in the next theorem. 

We next consider the special case when the covertexts are independent; i.e., QuiU2 = QuiQu2- We then 
have the following inner and outer bounds. 

Theorems Let QuiU2 = QuiQu2- Let TZ*^{Di, D2) be the closure of the convex hull of all (i?i,i?2) 
satisfying 



for some (?7i, Ti, U2,T2, Xi, X2,Y) G Sd^,D2^ and let 7^*„^(^)l, D2) be the closure of all ^2) satis- 
fying dl-® for some ([/i , Ti , C/2 , T2 , , X2 , F ) G Vd^ ,D2 ■ Then 



for all 5 > 0. 

The proof is given in Appendix iBl 

Remark 5 The cardinality of the alphabets of the auxiliary RVs Ti and T2 for TZ*^{Di, D2) can be bounded 
as \Ti\ < + 1, i = 1,2; see Appendix ICl 

Remark 6 In the simple case of independent covertexts QuiU2 = Qui Qu2 and parallel MAAC Wy\XiX2 = 
1X1^^2 (where 3^ = 3^2 x 3^2). the inner and outer bounds of Theorem [3] coincide and reduce to the 
capacity formula of two parallel single-user watermarking systems |[9l, ifTSll . 



Ri < I{Tr,Y\T2) - I{Ui-T{) 
R2 < I{T2;Y\Ti) - I{U2;T2) 
R1+R2 < I{Ti,T2;Y) - I{Ui-Ti) - I{U2-,T2) 



(4) 
(5) 
(6) 



ni^{D^,D2)^n{Di,D2)Qn*^t{Di + 5,D2+5) 
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Example Let the covertexts be independent binary sources with Ui = U2 = {0, 1} and Qui{Ui = 0) = 
0.05 and Qu2{U2 = 0) = 0.1. Let the MAAC be a binary additive channel with Xi = X2 = y = 
Z = {0, 1} and Y = Xi ® X2 ® Z, where Z is independent of {Xi,X2) with Pr(Z = 1) = 0.02 
and © denotes modulo 2 addition. Let Di = 0.45 and D2 = 0.4. Fig. |2] illustrates the numerically 
computed inner and outer regions of Theorems [T] and |2] (which coincide with the regions of Theorem [3] 
since Ui and U2 are independent). To compute 7?.*„ (0.45, 0.4), we only need to consider auxiliary RVs 
with alphabets |71| = jTil = 5. For comparison, we also plot two subsets of the region 7^*^^(0.45,0.4) 
by setting |7i| = |72| = 6 and |7I| = \T2\ = 7, respectively (recall that Theorem [3] does not give an 
upper bound on the alphabet sizes for T2 and T2 for the outer bound). It is seen that there exist noticeable 
gaps between 7^*^(0.45,0.4) and the numerically obtained subsets of 7^*^^(0.45,0.4). When computing 
the above regions, we quantized the unit interval using a step-size of resolution 0.1 to calculate the joint 
distributions. We can conclude that the obtained inner and outer bounds do not coincide, and furthermore, 
that in case there exists a finite upper bound on the auxiliary RV alphabet sizes for the outer region, this 
upper bound must be at least 7 for the binary problem. 



0.09 




Figure 2: The inner bound 7^*^(0.45, 0.4) for the Example and two subsets of 7^*^^(0.45, 0.4) obtained by 
setting |71| = |72| = 6 and |71| = [Til = 7- The obtained regions lie between the corresponding solid or 
dashed lines and the horizontal and vertical axes. 



3 Proof of Theorem [I] 

We first recall some notation and facts regarding strongly e-typicality. Let V = {Xi,X2, ...,Xm) be a 
superletter (a collection of RVs) taking values in a finite set V = A'l x ^2 x • • • x Xm and having joint 
distribution Py(xi, ...,Xm), which for simplicity we also denote by Py{v). Denote by T^^\v) or T^"^ 
the set of all strongly e-typical sequences S p. 326] with respect to the joint distribution Let 
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ly = {l,2,...,m}, and Ig ^ ly- We then let G = {Xg-^, Xg.^, Xg^^^^) € ^ be a "sub-superletter" 
corresponding to Iq such that gi € Iq- Let G, K, and L be sub-superletters of V such that Ig, Ik, 
II are disjoint, and let Pq, Pr and Pg\k be the marginal and conditional distributions induced by Py, 
respectively. Denote by Te'^\G) the projection of Te'^\v) to the coordinates of G. Given any k G /C", 
denote Ti"^(G|k) = {(G",k) G ri"^(G,i^)}. Clearly ri"^(G|k) = if k ^ T^'^\k). The following 
lemma (see, e.g., |4, pp. 342-343]) restates the well known exponential bounds for the cardinality of 
strongly typical sets. In the lemma r/ = 7y(e, n) is a generic positive term such that lim£_j.o \\m.n^^ ri{e, n) = 
0. 

Lemma 1 Pl 

1) For any < eo < 1 we have p'."')^(ri"''(G|k)|k) > 1 - eg for n sufficiently large. 



2) 2"(-f^(^)-'') < 7;^"^(i^) 



< 2"(^^(^)+'?). 



3) For any k G 7;^")(i^), 2-(^(G|^)-'') < 7;^"^(G|k 



< 2n(H(G|ii')+7y)_ 



Finally, we recall the Markov lemma for joint strong e-typicality. 

Lemma 2 (Markov lemma p^, p. 579]) Let G — > K — > L form a Markov chain in this order. For any 
< eo < 1 and (g,k) G ri"^(G, K), 



p(;^((g,k,L")GTW(G,i^,L) 
for n sufficiently large, independently of (g, k). 



k > 1-eo 



3.1 Outline of Proof 



It is enough to show that for given QuiU2' ^Y\XiX2' and any (i?i,i?2) G 7lin{Di, D2), there exists a 
sequence of codes {f^\ f'^\'^^'^^) such that Pi"-* ^ as n — )• 00 and for any 5 > 0, 



for n sufficiently large. Once this is proved, a standard subsequence diagonalization argument can be used 
to prove a similar statement with 5 = 0, which then directly implies the theorem. 

Fix {Pj-^^u^, Px^^\Ui_T-i_^ Pt2\U2^ ^X2\U2T2) such that /(C/i;Tj) > and the foUowmg are satisfied for some 
e' > 0, 

i?i </(ri;r2,y)-/(C/i;ri)-e', (7) 
R2 < I{T2;Ti,Y) - I{U2]T2) - e' , (8) 
+ </(Ti,T2;y)-/([/i,C/2;ri,T2)-e', (9) 
¥.[di{Ui,Xi)]<Di, i = 1,2. (10) 

We will choose /^"^ and 72"^ in a random manner. For e < 2max{d'"''^ ^maxy define 
P^ ^ Pr C/f )) > A + edr") , i = 1, 2. 



n 
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The technically challenging part of the proof is to show that for any < ei < gmaxjd™'^ ^maxy the 
probabilities pj"^ , P^"^^ , and ^2"^ , when averaged over the random choice of f[^^ and /g"^ , satisfy 

< ei, E[pf ^] < ei, E[P^''^] < ei 

for n sufficiently large. Then E{Pi"^ + P^"'^ + ^2"^} < 3ei, which guarantees that there exists at least one 
pair of codes /a^"^) such that P^""^ + P^""^ + P^"-^ < 3ei and hence p]"^ < 3ei, P^^"^ < 3ei, P^""^ < 3ei 
are simultaneously satisfied for n sufficiently large. Finally, it can be easily shown that p/"^ < 3ei implies 
for n sufficiently large that 



n 



3.2 Random Code Design 

In what follows, the strongly e-typical set t}""^ is defined under the joint distribution 

PU1U2T1T2X1X2Y = QUiU2Pti\UiPxi\UiTiPt2\U2Px2\U2T2^Y\X'^X2 (11) 

and all the marginal and conditional distributions, e.g., Pu2T2^ A/1IC/2T2' ^^^^ induced by the joint distri- 
bution. The parameter e, which is chosen to be sufficiently small, will be specified in the proof. 

Generation of codebooks. For i = 1, 2 and every wi G Wi, generate a codebook 

= {U{wi,l),ii{wi,2),...,ii{wi,Li)} 

with Li = 2'^[^(^''^»)+^'^l codewords such that each ii{wi, U) is independently selected with uniform distri- 
bution from the typical set 7^^"^ (Tj). Denote the entire codebook for Encoder i by C^*^ = {C^- where 
we recall that Mi = 2"^\ For each m and codeword ti(u)j, li) (1 < Wi < Mi,l < k < Li), generate a 
codeword Xj according to P^j^^ (xj|uj, tj). Denote the codebook of all the codewords Xj by 6^"^^ 

Encoder /}"^ Encoder /|"^ is the concatenation of a pre-encoder 93^"^ : Wi x — > TP and a 
mapping ^j"^ : ^/f x 7^" — > A'f . 

To define V'^"^ we need the following notation adopted from ifTTl . We introduce a conditional probability 

AW(Ui,ti) ^ Plj%^^um (("2,t2) : (U2,t2) G 7;(")(t^2T2|ui,ti) Ui,ti 

For n G (0, 1), let 

41iUi,n) ^ {(ui,ti) : A(")(ui,ti) > 1 - . 

By definition, we have ji"i(C/i, Ti) C t}''\Ui,Ti). 

We now describe the pre-encoding function (/?["^ = ip^^\wi,Ui) which maps every pair (ii;i,ui) to a 
codeword in C*-^^ C T{^. Given wi G {1, 2, Mi} and ui, c/?^"^ seeks the first codeword ti(tt;i, h) (if any) 
in Cwi such that {ui,ti{wi,li)) G T^} {Ui,Ti). If there is no such codeword, c/^^"-* outputs ti(r«i, 1). Next, 
for each output ti{wi, li) and ui, sends out the associated codeword xi(ii;i,ui) to the channel. Thus, 
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Encoder f^^^: Encoder 72"^ is the concatenation of a pre-encoder (^3"^ ■ ^2 x U2 — > TJ* and a 
mapping gf^ : x 7^" ^ X^. 
To define (^2"^ l^t 

, Ml 

<H'l2,t2) = ^ E (•11 (ui,^S"^(«;i,Ul)) G rW(C/iri|u2,t2)|u2,t2) . 

101 = 1 

Also, for V G (0, 1), define 

4ik.(f/2,r2) ^ {(U2,t2) : SW(U2,t2) > 1 - • 

By definition, it is seen that ^^^...(^/s, Ts) C %^''\U2, T2). 

We now describe the pre-encoding function 992"^^ = {w2^'^2) which maps every pair (ti;2)U2) to a 
codeword in C*^^^ C T2'- Given W2 € {1, 2, M2} and U2, (^2"^ seeks the first codeword t2(tt'2, h) (if any) 
in Cyj^ such that (u2, t2(w;2, 12)) £ J^!pllu,€iU2,T2). If there is no such codeword, v^g"'' outputs 12(1^2, !)• 
Next, for each output t2(iU2, h), 92^ sends out the associated codeword X2(f«2,ii2) to the channel. Thus, 
/2^"^(u;2,U2) =gf^ (u2,<^5"^(u;2,U2)). 

Decoder V'^"); Given y, V'^"^ seeks ii{wiji) € C(^) and t2(t?2,^2) G C^^) such that 

{ti{wi,h),t2{w2,T2),y) e 7;(")(ri,r2,y). 

If such a pair {ti{wi,li),t2{w2,l2)) exists for a unique (wi, ^2), then V'*^"^ outputs il;! and'u;2 as the decoded 
messages. If there is no such pair {'Wi,W2), or it is not unique, a decoding error is declared. Letting 
U{wi, k) = (^j-"^ {wi,Ui), it is easy to see that if there is a decoding error, then at least one of the following 
events occurs: 

1) Ei: {ti{wi,h)Mw2,l2),y) ^ %^''\Ti,T2,Y), 

2) E2: there exist l[ and w[ ^ wi and I2 (^2 may or may not be equal to I2) such that 

{ti{w[,i[)M'^2,i2),y) e 7;(")(ri,T2,y), 

3) E3: there exist I2 and W2 ^ W2 and I'l (l[ may or may not be equal to h) such that 

iti{w,,i[)Mw2,i2),y) e t}"\Ti,T2,y), 

or 

4) £'4: there exist l[ and lu^ 7^ and and W2 7^ W2 such that 

{ti{w[,i[)Mw2,i2),y) e r/")(ri,T2,F). 

In the following, we will bound the probabilities Pe , and P2 averaged over the random choice 
of all codes B^^\ B^^\ C^^\ and C^^\ To simplify the notation we abbreviate E^ci) g(2) (^(i) q(2) [ • ] as ]En[ • ]. 
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3.3 Bounding En [Fi"^] 

To analyze the average probability of error, we need the following lemmas. 

Lemma 3 For any wi G Wi, W2 € W2, and any eo, e € (0, 1), one can choose fi^v £ (0, 1) small enough 
such that 



PuX M"^(«)i,ui),ui,U2,4"^(u;2,U2)) G7;(")(ri,^i,C/2,T2) 



> 1-eo 



for n sufficiently large, where the expectation is taken with respect to the random codes C^^^ and C^'^\ 

The proof of Lemma [3] is very similar to the proof of the extended Markov lemma in 1 1 1 , Lemma 3] for 
correlated Gaussian sources and is hence omitted; readers may also refer to |[T4l Section 5.4.5]. 

Since the watermarks are independently and uniformly distributed, and by the symmetry of the code 
construction, we can assume without the loss of generality that some fixed wi G Wi and W2 G W2 are the 
transmitted watermarks. Thus we bound the probability of error as 

Pi") = Pr({^(")(y")/(u;i,t/;2)}) 

< Pr(^i)+Pr({^(")(y")/K,u;2)}|A^) (12) 

where Ai is the event 

: (tiK,/i),Ui,U2,t2(u;2,/2),xi,X2) ^7;(")(ri,C/i,C/2,r2,Xi,X2). 

Recall that ti^WiJi) = ip'f'\wi,Ui), i = 1,2. We also let ti^WiJ'-) and ti{w'i,l'^} be the l'--th codeword in 
the codebook . and C^/ , respectively. 
We then introduce the event 

^0 : (tiK,/i),Ui,U2,t2K,/2)) ^7;(")(Ti,C/i,?72,T2). 

Taking expectation in (fT2l) and using the union bound, we have 



MP^""^] <^nPi- {Ao) +EnPr (AMo) (EMI) + Y.^nPHEk\A'i) . (13) 



k=2 



It immediately follows from Lemma [3] that 

En Pr (^0) = Ecd) c(2) Pr (^0) < eo (14) 



for n sufficiently large, where we set eo = ei/7 for a given ei > throughout the proof. When Aq holds, 

' 



since xi and X2 are respectively drawn according to the conditional probabilities P^'l^^ ,p^(-|ui,ti) and 



P^'jjy^^j,^ (-[02,12), and y is drawn according to the conditional distribution W^"]j,^^^(-|xi,X2), it follows 
from two successive applications of Lemma[2]that 

EnPr(^il^g) <En[eo] = €0 (15) 
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and 

Eq Vt{Ei\A\) 

< En Pr ( { (<^S"Vi, f^r), C^r, U^, ^^"^(«^2, U^),fi''\wu f/f), /(")(«;2, U^),Y^) i T^''^]\a\ 

< En[eo]=eo (16) 

for n sufficiently large. It remains to bound Pr { E'^ | } for /c = 2, 3, 4. Using the union bound we 
write 

]KnPT{E2\Al) 

^ E Y^P^{{iTn<,l'l),Y^,T^in^2,l'2))^T!''\T^,T2,Y)]\A'i), (17) 

where TJ^{w[,l'i) is a RV uniformly drawn from Te^\Ti) which is independent of {T2{w2,l'2),y^) since 
w'l 7^ wi. Thus we have 



Pr({(rr(^,/;),y-,r2"(«;2,/^))G7;(")(ri,r2,y)} 



Al 



J2 E Pr{T,^{w2,l'2)=t2,Y-=y\A'i) 

(t2,y)eri"'(r2,y) tier.<"'(ri|t2,y) 

Pr (rf(u;;, = ti\T^{w2, Q = ta, = y, Al) 

E (7^2"(«^2, = t2, = y| A^) Pi {T^{w[,l[) = ti 

(t2 ,y) eri"' (Ta ,y ) ti er,<"' (Ti 1 12 ,y) 

Y: Pr(T,"(z.2,/2)=t2,y" = y|A5)l^^"^(^^l*^'^)l 



\Tr'{Ti)\ 



-in), 

{t2,y)€T}">iT2,Y) 
2n[H{n\T2,Y)+v] ^ 

^ ..i^m)-.] E Pr(T,-(^.2,/^)=t2,y" = yM5) 



(t2,y)eri"^(T2,y) 



2n[//(Ti)-,,] 

where the first inequality follows from Lemma[T] Recalling that r/ — as n — )• oo and e — > 0, we can make 
sure that 2r/ < e' — 4e by choosing e small enough and n large enough. Thus from ([17] ) 

EnFl{E2\A'i) < 2''^[Ri+IiUi;Ti)+4e-IiTr,T2,Y)+2v] 

< eo (19) 

for e sufficiently small and n sufficiently large, where ([T9l) follows from the assumption (O. Similarly we 
have 

EnPT { Al) <eo (20) 
for e small enough and n sufficiently large. We next bound 

EnPi {E^\Al) 

Li L2 

^ E E E Y^r[{iT^iw[,l[),T^iw'2,l'2),Yn^T}''HTuT2,Y)]\A'i), 
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where Tl^{w[,l[) and T^iwi^,!'^) are RVs independendy drawn from Te^'^Ti) and Te"'\T2) according to 
the uniform distribution, respectively. We have 



Pr({(TrK,ri),r2"K,/^),y") G7;(")(ri,r2,y)} 

= E E Pr{Y^ = y\At) 



yeri"' (Y) (ti ,t2)Gri"' (Ti ,T2 |y) 



Pr(rrK,ri) = ti,r2"K,/^) = t2|A5,y" = y) 



- = >'!^l)2n[/f(ri)-»7]2n[H(T2)-r?] 



<- 2-"[^(^l'^2;l')+-f(Ti;T2)-37?] 



and hence 



< 2"[-^i+-R2+/((7i;Ti)+/(C/2;T2)-7(Ti,T2;y)-/(Ti;T2)+8£+3r7] 

< 2"[-f^i+-f(^i'^2;Ti,T2)-/(Ti,T2;y)+£'] 

< eo (21) 

for n sufficiently large and e small enough (such that 8e + 3rj < e'), where the second inequality holds by the 
Markov chain relation Ti ^ Ui ^ U2 ^ T2 imposed in Definition |2l and the last inequality follows from 
the assumption Q. Finally, substituting (fT4l)-([T6l). ([T9l ). ( |20l ) and (EB into ([l3]) yields EQ[Pi"^] < 7eo = ei 
for e sufficiently small and n sufficiently large. 



(n)l 



3.4 Bounding Ef^[F. 

We only bound En [i^^"^] for i = 1, since the case i = 2 can be dealt with similarly. When (ui, xi(i(;i, ui)) G 
7;^")(C/i,Xi), 

idi(ui,xi(u;i,ui)) <E[di([/i,Xi)] + edr" <^i + edr" 
n 

for n sufficiently large, where the first inequality follows from the definition of strong typicality and the 
second inequahty follows from This means that if ^di [U]", f[''\Wi,U^)) > Di + erf^^', then we 
must have (^/f , /("^(M^i, ?7f )) ^ Te^"'\Ui,Xi) for n sufficiently large. Thus, we can bound 



< Pr((c/r,/f)(iyi,c/r)) ^7;(")([/i,Xi) 
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< Fv({^t'\Wi,U^),U^,U^,^'i'\W2,U^)) ^r}''HTi,Ui,U2,T2) 



(n). 



,{n) 



(22) 



Now taking expectation on both sides, the first term of (l22l ) is bounded by ^ by Lemma [3l and the sec- 
ond term is bounded by ^ for sufficiently large n by Lemma [T] This completes the proof of the bound 

IErj[-Pi"''] < ei for n sufficiently large. □ 



4 Concluding Remarks 

We have studied a multi-user information embedding system consisting of two information embedders and 
one joint decoder connected via a multiple-access attack channel. We have obtained an inner bound for 
the capacity region in a computable single-letter form. We also derived an outer bound for the capacity 
region, but in this case the auxiliary random variables involved in the region's characterization have no 
upper bounds on their alphabet's cardinality. Consequently, there may not exist an algorithm to compute 
the outer bound with arbitrary precision. We have also addressed the special case when the covertexts are 
independent of each other and inner and outer bounds for the capacity region of this simplified system are 
provided. Finally, we remark that using a similar technique inner and outer bounds are derived in lfT4l 
Chapter 5] for the capacity region of private multi-user embedding systems with quantization. 

Appendix 

A Proof of Theorem |2] 

The proof is a generalization of the proof of the converse in |[T5l| for a single-user embedding system. 

We need to show that any MAE code (/|"\ f2^\ 4'^^'^) with achievable rate pair R2) must satisfy 
([J)-© for some auxiliary RVs Ti and T2 with joint distribution PuiU2TiT2X-i_X2Y ^Z)i,D2- It follows 
from Fano's inequality that 

It is clear that e„ ^> if p]"^ — ^ and 

H{Wi\Y'') < H{Wi,W2\Y'')<nen, 
H{W2\Y'') < HiWi,W2\Y'') <n€n. 

Because Wi is uniformly drawn from the message set {1, 2, 2"^^ } and is independent of [/", we have 

nRi = H{Wi) = /(VFi; y") + F(W^i[y") < /(VFi; F") - /(VFi; ) +ne„. 

=0 
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Hence we can write 

i{Wr,Y^)-i{Wv,u^) 

n 

[l{Wr,Y,\Yt')-I{W,-U,u\Ul^^,] 



(a) 



k=l 



[H(Y,\Yt') - H{Yk\W,,Yt\ f/iVi) - I(Xk; ^iVil^i' n'"') 

k=l 

n 

Y [h{Y,\Y,'-') - H{Y,\WuY,'-\ C/iVi) - I{U,,; Y^'m, C/iVJ 

k=l 

-i7(C/i,|C/iVi) + HiU^k\W,, f/iVi)] 

n 

Y [H{Yk\Y>'-') - HiYk\Wi,Y,^-\U^^, 

k=l 

-H{Uik) + H{Uik\Wi,Y^-\Ulk_,^) 

n 



,k+l, 



k=l 



(d) 



k=l 

n 



< Y [l{W2,U^,,+^,Yt\Y,;W,,U^^,^,,Yt') - I{Uik;W,^^^^^ 

k=l 

n 

^[-^(-Z^2A:, Yk] Lik) — I{Uik; Lik)] 



k=l 



Where in (a) Y^'-' ^ (Fi, Fs, ^-i) and U^^^^^ ^ {Ui ,fc+i) ^i,fc+2, •••) (b) follows from the 

"summation by parts" identity 13] Lemma 7], (c) holds since the source C/i is memoryless, in (d) U2f^_^i — 
{U2,k+i,U2,k+2,:;U2,n), and in (e) Lik = f^iVi) ^"^^ ^2fc = {W2,Y,^~\U^^,^^^). Hence 

we obtain the bound 



1 " 

^1 < - V[/(Life;L2fe,n) -/(C/ifc;Lifc)] + e„. 
n ^-^ 



(23) 



k=\ 



Similarly, we can show that 



1 " 

R2 < - Y^I{L2k]Lik,Yk)-I{U2k\L2k)]+en- 



(24) 



k=l 



To bound the sum of the rates, we write 



n{Ri + R2) = H{Wi,W2) = I{Wi,W2]Y'^) + H{Wi,W2\Y'^) 
< I{Wi,W2;Y'') - I{Wi,W2\ C/r, U^) +nen 



(25) 
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and 

i{w,, W2; y") - i{w,, W2; u^, u^) 

n 

fc=i 

n 
k=l 

-H{Uik, U2k\U^^k+i, U^,k+i) + HiUik, U2km,W2, ^iVi' U2,k+i)] 

n 

= [H{Yk\Yt^)-H{Y,\W^,Ulk^,,Yt\W2,Ul^+^)-I{U^k,U2k^^^^^ 

k=l 

-H{Uik, U2k) + H{Uik. U2k\Wi,W2, U^^k+i, U^,k+i)] 

n 

= Y [H{Yk\Yt') - H{Yk\W,,U^^,^„Y,'-\W2,Ul,^^) 

k=l 

-H{Uik, U2k) + H{Uik, U2k\Wi,W2, U^,k+i,Yt^) 

n 

< Y [H{Yk) - HiYk\Wi,U^^,^„W2,Ulf,^„Y,'-') - I{Uik,U2k;Lik,L2k) 

k=l 
n 

= Y [l{Yk;Wi,Uli,^„W2,U^^k+i,Yt') - HUik,U2k;L,k,L2k) 
fc=i 

n 

= ^[-^(yfc; Lik,L2k) — I{Uik-, U2k', Lik,L2k)], 

k=l 

which implies 

1 " 

R1 + R2 <-y^[I{Lik,L2k;Yk)-IiUik,U2k]L-^k,L2k)]+en- (26) 

k=l 

We next introduce a time-sharing RV to simplify the bounds (1231 ). (l24l ). and (l26l ) using a single-letter 
characterization. Define a RV V with alphabet {1, 2, n} and distribution Pv'(t') = 1/n. We next intro- 
duce RVs Ui and [/2 such that 

Pr(C7i =ui,U2= U2) = Pr(?7ifc = ui,U2k = U2) = QuiU2(.ui,U2) 

for all {ui,U2) GUi X U2, which are independent of V. Furthermore, we define new RVs Li, L2, Xi, X2, 
and Y by 

Pr(Li =h,L2 = l2,Xi =xi,X2=X2,Y = y\V = k) 
= Pr(Lifc = li,L2k = h-,Xik = xi,X2k = X2,Yk = y) 
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for all {h,l2,xi,X2,y) G £1 x £2 x Afi x A2 x 3^. It follows that 

- Y^[I{Lik;L2k, Yk) - /(C/ife; Life)] 
" fe=i 

= IiW,L2,Y\V)-I{Ui;L^\V) 

= H{Li\V) - H{L,\L2,Y,V) - H{Ui\V) + H{Ui\Li,V) 

< H{Li) - HiL,\L2,Y, V) - H{Ui) + H{Ui\Lu V) 
= I{Li-L2,Y,V)-I{Ui-Li,V) 

< I{L,,V;L2,Y,V)-I{Ui;Li,V) 

I{Tv,T2,Y) - I{Ti;Ui) 

where (a) holds since conditioning reduces entropy and Ui is independent of V, and in (b) Ti = (Li , V) 
and T2 = (L2, V). This shows that 

Ri < I{Ti-T2,Y) - /(Ti; Ui) + e„. (27) 

By a similar argument, we can show 

R2 < I{T2 ; Ti , y ) - I{T2 ; C/2) + e„ (28) 

and 

Ri + R2< I{Ti,T2;Y) - I{Ui, C/2; Ti, Ts) + e„. (29) 

For such RVs ( C/i , C/2 , ?! , T2 , Xi , X2 , F ) , it can be readily seen that the Markov chain relation (Ui,U2,Ti,T2) 
(Xi,X2) holds. In fact, 

Pr(y = y\Ui = ui,U2 = U2,Ti = h = {li,k),T2 = ^2 = {l2,k),Xi = xi, X2 = X2) 
= Pr(y = y\Ui = ui,U2 = U2,Li = li,L2 = l2,Xi = xi,X2 = X2,V = k) 
= Pr{Yk = y\Uik = ui, U2k = U2,Lik = li,L2k = h^Xik = xi,X2k = X2) 
= Pr(yfe = y\Xik = xi,X2k = X2) 

Next we bound the distortions W\di{Ui,Xi)]. Since (i?i,i?2) is achievable under the sequence of codes 
(/i"^ , , V'*^"^ )' this impUes that for any ^ > and all n large enough, we have 

1 1 ^' 



= - Pr(^7r = u,,Xf =x,)d,(u„x,) 

1 " 



n 

k=iurxxr- 



= "^PviV = k) "Y Pr{Uik = Uik, Xik = Xik)di{uik, Xik) 
fe=l UiXXi 
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n 

= Z]^^(^ = ^) Yl P^iUi = Ui,Xi = X^\V = k)d^{u^,Xi) 
k=l UiXXi 
n 

= ^ ^ Vr{Ui=Ui,Xi = Xi,V = k)di{ui,Xi) 

k=l UiXXi 

= ^ Pu,x,iui,Xi)di{ui,Xi). 

UiXXi 

Thus we obtained that E[di{Ui, Xi)] < Di + 5 for i = I, 2. Combined with (|T7])-(|29l) and recalhng that 
liuin^ao Cn = and that Tl{Di,D2) is closed, we conclude that Tl{Di,D2) C lZout{Di + 6, D2 + 6) as 
claimed. □ 



B Proof of Theorem m 

The forward part (achievability) is a consequence of Theorem [T] since {Ui,Ti) and (C/2) ^2) are independent 
and hence /(Ti; Ts, = /(Ti; FlTs), /(Ts; Ti, F) = /(Ts; y|ri), and I(?7i, C/2; Ti, Ts) = /([/i; Ti) + 
/(C/2; 22)- To prove the converse part, we need to sharpen the bounds in the last proof. We start from 

I{Wi;Y^) - I{Wi;U^) 

n 

k=l 
n 

fc=i 

n 
k=l 

n 

k=l 
n 

< Y [/iVi, Fi'^-i; y,|VF2, US,k+i,Yt') - I{Uik; WuYt\ C/iVi)" 

k=l 
n 

= Y^I{L,k;Yk\L2k)-I{Uik;Lik)] 

k=l 

where (a) follows since {Wi, independent of {W2, Ul^k+i)' ^^'^ ^^^^ equality we still 

let Life = {Wi,Y^-^, Ul^+i) and L2fe = {W2, Y^'^ , C^2"fc+i)- Thus, using Fano's inequality we have 

1 " 

Ri<- y\I{Lik;Yk\L2k) - HUik-, Lik)] + en. 
k=i 

Similarly we can obtain 

1 " 

R2<- y\I{L2k\Yk\Lik) - I{U2k; L2k)] + en- 
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To bound the sum of the rates, we have 



n{Ri+R2) = H{Wi,W2) = I{Wi,W2;Y'') + H{Wi,W2\Y'') 
< I{Wu W2; y") - I{Wv, U^) - I{W2; U^) + nen 



and 



/(w^i, W2\ r") - u'^) - m2. U^) 

\l{Wv,Yk\Y^^-')+I{W2;Yk\Wi,Y,^-')-I{Wi;Uik\U^^k+i)-I^ 



E 

k=l 
n 

E 

fc=i 

n 

E 

k=l 

n 

E 

k=l 



E 

k=l 



= E 

k=l 



(a) 



k=l 



E 

k=l 



rk-l 



k-U 



+H{Yk\Wi,Y,'-') - H{Yk\Wi,W2,Y,'-\ C/2V1) " ^i^k; U^]k+i\Wi, W2,Y, 
-H{Uik\U{\^,) + H{U^u\Wi, Ul,^,) - H{U2k\Ulk+i) + H{U2k\W2, C/2V1)] 

-//(C/ifc) + H{U,k\Wi,Ul]k+i) - H{U2k) + H{U2k\Wi,W2, C/2V1)] 



HiY,\Yt')-H{Yk\W,,Yr\U]^,k+i) 



k — l TTTl 



+H{Yk\Wi,Y,''') - H{Yk\W,,W2,Y,'-\U^^k+i) 

-H{Uik) + H{Uik\Wi,U^^k+i,Yi~') - H{U2k) + H{U2km, ^2, C/2V1' ^i"""') 
/(n; Wi, C/iViln'"') + m\ W2, C/2Vil^i' ^1'"') 
-7(C/ifc; W^i, Ulk+i,Y^-^) - I{U2k; W2, U^,k+i,Y,'-')_ 
HiWuK^,^,\Yt')-H{W,,U[:k+i\yi-\Yk) 

+HiW2,U^,k+i\Wi,Y,^-') - H{W2,U^^k+i\Wi,Y,'-\Yk) 

- /(C/i,; W,, C/iVi. ^1'"') - mk-, W2, C/2V1' 

+H{W2, ^sVil^i' f^iVi' n'"') - HiW2, U^,k+i\Wi,U^,k+i,Yt\Yk) 

- I{U,k; Wi, ?7iVi, Yt') - I{U2k; W2, U^,k+i,Yt')_ 

HiWu C/iVi- ^2, U^,k+i\Yt') - H{WuUl],^„ W2, US,k+i\Yt\Y,) 

- mk;Wi,U^^f,^,,Y,^-')-I{U2k;W2,U^,k+i,Yt') 
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n 

k=l 

n 

k=l 

-I{U2k;W2,U^,,^„Yt') 

n 

= [l{Wi, Ul,_,„W2, U^,k+i,Yt';Yk) - I{Uik; Wi, f/iVi' n'"') - IiU2k;W2, U^,k+i,Yt') 

k=l 
n 

= ^ [I{Lik, L2k', Yk) - I{Uik] Lik) — I{U2k] L2k)] 

k=l 

where (a) holds since {Wi^U^^^^) is independent of (W2, f/a^fc+i) and Lifc = [Wi^Y^'^ ^U^^^^) and 
L2k — {W2,Y^~^ , ^2k+i) ^^'^ ^a^t equality. The above implies 

1 " 

i?l + i?2 < — / ^ [/(Lifc, L2fc; Ifc) - I{Uik]Lik) - I{U2k] L2k)] + en- 
k=l 

The rest of the proof proceeds the same way as the proof of Theorem |2l □ 

C Upper Bounds on |7I| for 7^■„(L>l, D2) and 7^„,(L'l, D2) 

We only bound the cardinality of 71 and 72 for the region 7^*„(Z)i, D2)- The bounds for |7i| and [Til for the 
region 7^j„(L'i, D2) can be derived in a similar manner. We will need the following support lemma, which 
is based on Caratheodory's theorem on the convex hull of a set in a finite-dimensional vector space. 

Lemma 4 (fT. Support lemma, p. 311]) Let V{X) be the set of distributions defined on a finite set X (rep- 
resented as the probability simplex in IRI'^I) and let fj,j = 1, 2, A; be real-valued continuous functions on 
V{X). For any probability measure on the Borel cr-algebra of V{X), there exist k elements Pi, P2, ■■■,Pk 
of V{X) and k non-negative reals ai, 02, ...0^ with Ym=i Q^i = 1 such that for every j = 1, 2, k 

JV{X) ^ 

Using this lemma, we will show that for any given PxiTi\Ui and Px^t2\U2' there exists a RV Ti with 
|7i I < [^1 1 [A^i I + 1 only depending on Ui and Xi such that the following hold 

I{fi-Y\T2) - I{Ui-fi) = I{Ti-Y\T2) - I{Ui;Ti) (31) 
I{T2-^Y\fi) - I{U2;T2) = I{T2;Y\Ti) - I{U2;T2) (32) 
I{fi,T2;Y) - I{Ui;fi) - I{U2;T2) = /(Ti, Ta; Y) - /([/i; Ti) - /(C/2; Ta), (33) 

and that the expectation of the distortion between Ui and Xi is preserved when Ti is replaced by Ti . Note 
that the upper bound on |7i| does not depend on iTil- 
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We first rewrite 

/(Ti;y|T2)-/(C/i;ri) = H{Y\T2) - H{Y\Ti,T2) - H{Ui) + H{Ui\n), 
I{T2;Y\Ti) - I{U2;T2) = H{Y\T^) - H{Y\Ti,T2) - I{U2;T2), 

and 

I{Ti,T2;Y) - I{Ui-Ti) - /(C/aiTa) = H{Y) - H{Y\Ti,T2) - H{Ui) + H{Ui\Ti) - I{U2;T2). 
Recall that tlie joint distribution of {Ui,U2,T2,T2, Xi, X2, Y) can be factorized as 

PU1T1U2T2X1X2Y = QUiU2PTiXi\UiPt2X2\U2^Y\XiX2- 

We note that there exists a Markov chain {Ti,Xi) Ui ^ U2 ^ {T2, X2). Writing 

PUinU2T2XiX2Y = PTiPuiXi\TiPu2\UiPt2X2\U2Wy\XiX2, 

and noting that Pjj^p^, Pt2X2\U2 ^'^^ ^y\XiX2 fixed, to apply the support lemma, we need m — 1 
functions to preserve the joint distribution of {Ui,Xi) (see (l34l) below), where m = Specifically, 
we define the following real-valued continuous functions of distribution PuiXi\Ti{'i 'l^i) on Ui x Xi for 
fixed ti G 71, 

/«i,xi(^ViXi|Ti(-> -l^l)) - PuiXi\Tx{ui,Xi\ti) 

for all {u\,x\) gUixXi except one pair {ui,xi). Furthermore, we define real-valued continuous functions 

U{Pu^x,\Td-r\ti)) = -Hp{Y\T,=h,T2) + Hp{Ui\Ti = h), 
U+i{Pu,x,\Td-r\ti)) = Hp{Y\Ti = h)-Hp{Y\Ti = h,T2), 

where the entropies are taken under the joint distribution induced by PjjiXilTii'j 'l^i)- According to the 
support lemma, there must exist a new RV Ti (jointly distributed with {Ui,Xi)) with alphabet size |Ti | = 
m + 1 = lA^il + 1 such that the expectation of fi,i = 1, 2, m+1, with respect to Pt^ can be expressed 
in terms of the convex combination of m + 1 points; i.e., 

PUiXi{ui,Xi) = J2 PTiitl)fui,xi{PUiXi\Ti{-,-\tl)) 

tieTi 

= E ^f,(*'i)/-i,-i(^t/iX,|fi(-'-|*'i))' (34) 

-H{Y\Ti,T2) + HiUi\Ti) = ^TAh)fm{Pu,x,\TA-r\ti)) 

tieTi 

= 5Z ^fi(*i)/™ (^c/iXi|fi(-'-|*i)) 

tiefi 

= -H{Y\fi,T2)+H{Ui\fi), 

H{Y\Ti)-HiY\TuT2) = PT^{ti)fm+i{Pu,x,\TA-r\ti)) 

tieTi 

= H{Y\fi)-H{Y\fi,T2). 
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This implies that (l3TI)-(l33l) hold. It should be point out that this RV Ti maintains the prescribed distortion 
level, since Pu-^^Xi{ui,xi) is preserved. Similarly, for any given PxiTi\Ui ^^id Px2T2\U2' show that 

there exists a RV T2 with [72| < |^2| |'^2| + 1 only depending on U2 and X2 such that 

/(ri;y|f2)-/(c/i;ri) = /(ri;y|r2) -/(c/i;ri) (35) 

/(f2;y|ri)-/(C/2;f2) = I{T2;Y\Ti) - I{U2;T2) (36) 
I{Ti,f2;Y) - I{Ur,Ti) - I{U2;f2) = /(Ti, r2; F) - /(C/i; Ti) - /(C/2; ^2), (37) 

and the distortion constraint between U2 and X2 is preserved. Thus we conclude that the cardinality of Ti 
can be bounded by iZ^jjlAil + 1, i = 1, 2. 

Finally, we remark that the support lemma cannot be straightforwardly used to bound the cardinality 
for 71 and T2 for the region lZout{Di, D2) and Tl*,^^{Di, D2). For example, to bound the cardinality of 71 
for TZout{Di, D2), we need [Z^i||^/2||'^i|I'^2||'72| — 1 real- valued continuous functions to preserve the joint 
distribution of {Ui, U2,T2, Xi, X2). Therefore, we may need |Z^2| I'^il |'^2l|72| + 1 letters and this upper 
bound depends on |72|. □ 
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